Currently I have made request_chain of type Arc<Mutex<RequestChain>>. This is because reqwest::Request is not Clone but Client and ClientBuilder are Clone. Do you think it should be done this way? The alternative would probably be to update RequestChain to be Chain<OurOwnRequestType, Status>. When OurOwnRequestType implements Copy we probably would also facilitate the Extism plugins since copying data from/to the plugins is necessary.

Yeah, that might make sense. I don't know. After all, reqwest::Request is an implementation detail.
From what I can see, the only field in reqwest::Request that isn't Clone is the Body, and that's because reqwest::Body also has a Stream variant (which is behind a feature flag)
https://docs.rs/reqwest/latest/src/reqwest/async_impl/body.rs.html#18-20
That means that, yeah, we can wrap reqwest::Request and make it Clone, but we'd lose the ability to use Stream, which I don't think we're planning to use.

There is no response chain yet. Should I try to implement one in this PR?

I added a comment for that. Maybe there's a way, but if it makes things too complicated, we can leave it out for now.

This PR contains nothing related to Extism yet. I think it would be awesome if WASM plugins could be provided on the command line. So this would probably be lychee-bin loading the plugins and passing it to lychee-lib. Should I try add this in this PR?

For now, we just need to make sure that Extism works with the design we have in mind. We can create a separate branch+PR, which is based on this branch and adds one plugin based on Extism as a proof-of-concept.

@HU90m, since this also affects our Architecture discussion, it would be great if you could also take a look. 😃

For some background, @thomas-zahner and me talked about a simple plugin-system offline and that it would be based on Extism. This first step is to work on the trait system, which will allow plugins in the future.

@thomas-zahner can probably give a better answer here, but the idea is that at any point of the chain, a step can decide to stop the execution and return a final conclusion without passing the request on to the next step.
So filters would be part of the request chain, yes. For example the exclude filter could send a ChainResult::Done in case a URL got excluded.

Every plugin in the chain would be able to make network requests or access files, so yes, they will be able to access I/O, but I'm not sure what is the relation to ChainResult::Done in that case. (If you meant that Done would indicate that an I/O operation is done, then the answer is no, this would not be what it means. Instead, this is done with async/await, where an I/O operation will be .awaited and the runtime will take care of the rest.

@HU90m Yes filters are a good example of a Chainable which return a ChainResult::Done. But it can be anything that wants to end the chain and make a status out of the request. I'm currently working on making the actual final HTTP request to be also a chainable which returns ChainResult::Done.

Looks fine.
We got two more action items:

• Simple plugin system #1389 (comment)
• Check that the trait can be used in combination with extism.

@mre See my latest commit for the try_clone().unwrap().

I've created an extism-prototype branch for a first simple but functional Extism-plugin system. Feel free to give any feedback.

I think we want to make the Chainable trait and ChainResult public, so that library users can add their own handlers.
Created a PR on top of your branch for that and added a bit of documentation, since this will be a central part of lychee going forward.

thomas-zahner#3

You can review and merge into your branch if it looks fine. After that, let's get this shipped. 😅 Sorry for the long review delay.